A talk by Arif Wider & Emily Gorcenski presented at NDC Porto '20
Abstract:
It is already challenging to transition a machine learning model or AI system from the research space to production, and maintaining that system alongside ever-changing data is an even greater challenge. In software engineering, Continuous Delivery practices have been developed to ensure that developers can adapt, maintain, and update software and systems cheaply and quickly, enabling release cycles on the scale of hours or days instead of weeks or months. Nevertheless, in the data science world Continuous Delivery is rarely been applied holistically.
This is partly due to different workflows: data scientists regularly work on whole sets of hypotheses, whereas software engineers work more linearly even when evaluating multiple implementation alternatives. Therefore, existing software engineering practices cannot be applied as-is to machine learning projects. Learn how we used our expertise in both fields to adapt practices and tools to allow for Continuous Intelligence–the practice of delivering AI applications continuously.
Artificial intelligence and machine learning are currently all the rage. Every organisation is trying to jump on this bandwagon and cash in on their data reserves. At ThoughtWorks, we’d agree that this tech has huge potential — but as with all things, realising value depends on understanding how best to use it.
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...Dr. Arif Wider
A talk presented by Max Schultze from Zalando and Arif Wider from ThoughtWorks at NDC Oslo 2020.
Abstract:
The Data Lake paradigm is often considered the scalable successor of the more curated Data Warehouse approach when it comes to democratization of data. However, many who went out to build a centralized Data Lake came out with a data swamp of unclear responsibilities, a lack of data ownership, and sub-par data availability.
At Zalando - europe’s biggest online fashion retailer - we realised that accessibility and availability at scale can only be guaranteed when moving more responsibilities to those who pick up the data and have the respective domain knowledge - the data owners - while keeping only data governance and metadata information central. Such a decentralized and domain focused approach has recently been coined a Data Mesh.
The Data Mesh paradigm promotes the concept of Data Products which go beyond sharing of files and towards guarantees of quality and acknowledgement of data ownership.
This talk will take you on a journey of how we went from a centralized Data Lake to embrace a distributed Data Mesh architecture and will outline the ongoing efforts to make creation of data products as simple as applying a template.
Watch full webinar here: https://buff.ly/2XXbNB7
What started to evolve as the most agile and real-time enterprise data fabric, Data Virtualization is proving to go beyond its initial promise and is becoming one of the most important enterprise big data fabrics.
Attend this session to learn:
*What data virtualization really is
*How it differs from other enterprise data integration technologies
*Why data virtualization is finding enterprise wide deployment inside some of the largest organizations
3 Reasons Data Virtualization Matters in Your PortfolioDenodo
Watch the full session on-demand here: https://goo.gl/upxC5W
Real-Time Analytics for Big Data, Cloud & Self-Service BI
The world of data is only becoming distributed. Privacy, regulations, and the need for real-time decisions are challenging organizations’ legacy information strategy. This webinar will include an expert panel discussion on Logical Data Warehouse, Universal Semantic Layer, and Real-time Analytics by Paul Moxon (VP of Data Architectures), Pablo Alvarez (Director of Product Management), and Alberto Pan (CTO).
Attend and learn:
• The major challenges of legacy information strategies.
• How data virtualization can help you overcome these challenges.
• Strategies for enabling agile data management and analytics.
Unlock Your Data for ML & AI using Data VirtualizationDenodo
How Denodo Complement’s Logical Data Lake in Cloud
● Denodo does not substitute data warehouses, data lakes,
ETLs...
● Denodo enables the use of all together plus other data
sources
○ In a logical data warehouse
○ In a logical data lake
○ They are very similar, the only difference is in the main
objective
● There are also use cases where Denodo can be used as data
source in a ETL flow
Simplifying Cloud Architectures with Data VirtualizationDenodo
Watch here: https://bit.ly/2yxLo6f
Moving applications and data to the Cloud is a priority for many organizations. The benefits - in terms of flexibility, agility, and cost savings - are driving Cloud adoption. However, the journey to the Cloud is not as easy as many people think. The process of moving application and data to the Cloud is challenging and can entail widespread disruption across the organization if not carefully managed. Even when systems are migrated to the Cloud, the resultant hybrid or multi-Cloud architecture is more complex for users to navigate, making it harder for them to get the data that they need to do their jobs.
Data Virtualization can help organizations at all stages of their journey to the Cloud - during migration and also in the resultant hybrid or multi-Cloud architectures. Attend this webinar to learn how Data Virtualization can:
- Help organizations manage risk and minimize the disruption caused as systems are moved to the Cloud
- Provide a single point of access for data that is both on-premise and in the Cloud, making it easier for users to find and access the data that they need
- Provide a security layer to protect and manage your data when it's distributed across hybrid or multi-Cloud architectures
Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)Denodo
Watch full webinar here: https://bit.ly/3dudL6u
It's not if you move to the cloud, but when. Most organisations are well underway with migrating applications and data to the cloud. In fact, most organisations - whether they realise it or not - have a multi-cloud strategy. Single, hybrid, or multi-cloud…the potential benefits are huge - flexibility, agility, cost savings, scaling on-demand, etc. However, the challenges can be just as large and daunting. A poorly managed migration to the cloud can leave users frustrated at their inability to get to the data that they need and IT scrambling to cobble together a solution.
In this session, we will look at the challenges facing data management teams as they migrate to cloud and multi-cloud architectures. We will show how the Denodo Platform can:
- Reduce the risk and minimise the disruption of migrating to the cloud.
- Make it easier and quicker for users to find the data that they need - wherever it is located.
- Provide a uniform security layer that spans hybrid and multi-cloud environments.
Watch this webinar in full here: https://buff.ly/2MVTKqL
Self-Service BI promises to remove the bottleneck that exists between IT and business users. The truth is, if data is handed over to a wide range of data consumers without proper guardrails in place, it can result in data anarchy.
Attend this session to learn why data virtualization:
• Is a must for implementing the right self-service BI
• Makes self-service BI useful for every business user
• Accelerates any self-service BI initiative
Artificial intelligence and machine learning are currently all the rage. Every organisation is trying to jump on this bandwagon and cash in on their data reserves. At ThoughtWorks, we’d agree that this tech has huge potential — but as with all things, realising value depends on understanding how best to use it.
Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...Dr. Arif Wider
A talk presented by Max Schultze from Zalando and Arif Wider from ThoughtWorks at NDC Oslo 2020.
Abstract:
The Data Lake paradigm is often considered the scalable successor of the more curated Data Warehouse approach when it comes to democratization of data. However, many who went out to build a centralized Data Lake came out with a data swamp of unclear responsibilities, a lack of data ownership, and sub-par data availability.
At Zalando - europe’s biggest online fashion retailer - we realised that accessibility and availability at scale can only be guaranteed when moving more responsibilities to those who pick up the data and have the respective domain knowledge - the data owners - while keeping only data governance and metadata information central. Such a decentralized and domain focused approach has recently been coined a Data Mesh.
The Data Mesh paradigm promotes the concept of Data Products which go beyond sharing of files and towards guarantees of quality and acknowledgement of data ownership.
This talk will take you on a journey of how we went from a centralized Data Lake to embrace a distributed Data Mesh architecture and will outline the ongoing efforts to make creation of data products as simple as applying a template.
Watch full webinar here: https://buff.ly/2XXbNB7
What started to evolve as the most agile and real-time enterprise data fabric, Data Virtualization is proving to go beyond its initial promise and is becoming one of the most important enterprise big data fabrics.
Attend this session to learn:
*What data virtualization really is
*How it differs from other enterprise data integration technologies
*Why data virtualization is finding enterprise wide deployment inside some of the largest organizations
3 Reasons Data Virtualization Matters in Your PortfolioDenodo
Watch the full session on-demand here: https://goo.gl/upxC5W
Real-Time Analytics for Big Data, Cloud & Self-Service BI
The world of data is only becoming distributed. Privacy, regulations, and the need for real-time decisions are challenging organizations’ legacy information strategy. This webinar will include an expert panel discussion on Logical Data Warehouse, Universal Semantic Layer, and Real-time Analytics by Paul Moxon (VP of Data Architectures), Pablo Alvarez (Director of Product Management), and Alberto Pan (CTO).
Attend and learn:
• The major challenges of legacy information strategies.
• How data virtualization can help you overcome these challenges.
• Strategies for enabling agile data management and analytics.
Unlock Your Data for ML & AI using Data VirtualizationDenodo
How Denodo Complement’s Logical Data Lake in Cloud
● Denodo does not substitute data warehouses, data lakes,
ETLs...
● Denodo enables the use of all together plus other data
sources
○ In a logical data warehouse
○ In a logical data lake
○ They are very similar, the only difference is in the main
objective
● There are also use cases where Denodo can be used as data
source in a ETL flow
Simplifying Cloud Architectures with Data VirtualizationDenodo
Watch here: https://bit.ly/2yxLo6f
Moving applications and data to the Cloud is a priority for many organizations. The benefits - in terms of flexibility, agility, and cost savings - are driving Cloud adoption. However, the journey to the Cloud is not as easy as many people think. The process of moving application and data to the Cloud is challenging and can entail widespread disruption across the organization if not carefully managed. Even when systems are migrated to the Cloud, the resultant hybrid or multi-Cloud architecture is more complex for users to navigate, making it harder for them to get the data that they need to do their jobs.
Data Virtualization can help organizations at all stages of their journey to the Cloud - during migration and also in the resultant hybrid or multi-Cloud architectures. Attend this webinar to learn how Data Virtualization can:
- Help organizations manage risk and minimize the disruption caused as systems are moved to the Cloud
- Provide a single point of access for data that is both on-premise and in the Cloud, making it easier for users to find and access the data that they need
- Provide a security layer to protect and manage your data when it's distributed across hybrid or multi-Cloud architectures
Simplifying Your Cloud Architecture with a Logical Data Fabric (APAC)Denodo
Watch full webinar here: https://bit.ly/3dudL6u
It's not if you move to the cloud, but when. Most organisations are well underway with migrating applications and data to the cloud. In fact, most organisations - whether they realise it or not - have a multi-cloud strategy. Single, hybrid, or multi-cloud…the potential benefits are huge - flexibility, agility, cost savings, scaling on-demand, etc. However, the challenges can be just as large and daunting. A poorly managed migration to the cloud can leave users frustrated at their inability to get to the data that they need and IT scrambling to cobble together a solution.
In this session, we will look at the challenges facing data management teams as they migrate to cloud and multi-cloud architectures. We will show how the Denodo Platform can:
- Reduce the risk and minimise the disruption of migrating to the cloud.
- Make it easier and quicker for users to find the data that they need - wherever it is located.
- Provide a uniform security layer that spans hybrid and multi-cloud environments.
Watch this webinar in full here: https://buff.ly/2MVTKqL
Self-Service BI promises to remove the bottleneck that exists between IT and business users. The truth is, if data is handed over to a wide range of data consumers without proper guardrails in place, it can result in data anarchy.
Attend this session to learn why data virtualization:
• Is a must for implementing the right self-service BI
• Makes self-service BI useful for every business user
• Accelerates any self-service BI initiative
The Role of the Logical Data Fabric in a Unified Platform for Modern AnalyticsDenodo
Watch full webinar here: https://bit.ly/3FHKalT
Given the growing demand for analytics and the need for organizations to advance beyond dashboards to self-service analytics and more sophisticated algorithms like machine learning (ML), enterprises are moving towards a unified environment for data and analytics. What is the best approach to accomplish this unification?
In TDWI’s recent Best Practice Report, Unified Platforms for Modern Analytics, written by Fern Halper, TDWI VP Research, Senior Research Director for Advanced Analytics, adoption, use, challenges, architectures, and best practices for unified platforms for modern analytics is explored. One of the approaches for unification outlined in the report is a data fabric approach.
Join us for a webinar with our Director of Product Marketing, Robin Tandon, where he will discuss the role of the logical data fabric in a unified platform for modern analytics, focusing on several of the key findings outlined in this report. He will share insights and use case examples that demonstrate how a properly implemented logical data fabric is the most suitable approach for Unified Data Platforms across enterprises and organizations.
Watch on-demand & Learn:
- The benefits of a unified platform and its ability to capture diverse & emerging data types and how to support high performance and scalable solutions.
- The role of an enhanced AI driven data catalog and its implications towards the findings in the best practice report.
- Implications of a logical data fabric as it relates to several of the recommendations outlined in the report.
Minimizing the Complexities of Machine Learning with Data VirtualizationDenodo
Watch full webinar here: https://buff.ly/309CZ1Y
Advanced data science techniques, like machine learning, have proven an extremely useful tool to derive valuable insights from existing data. Platforms like Spark, and complex libraries for R, Python and Scala put advanced techniques at the fingertips of the data scientists. However, these data scientists spent most of their time looking for the right data and massaging it into a usable format. Data virtualization offers a new alternative to address these issues in a more efficient and agile way.
Attend this webinar and learn:
*How data virtualization can accelerate data acquisition and massaging, providing the data scientist with a powerful tool to complement their practice
*How popular tools from the data science ecosystem: Spark, Python, Zeppelin, Jupyter, etc. integrate with Denodo
*How you can use the Denodo Platform with large data volumes in an efficient way
*About the success McCormick has had as a result of seasoning the Machine Learning and Blockchain Landscape with data virtualization
Big Data Fabric for At-Scale Real-Time Analysis by Edwin RobbinsData Con LA
Abstract:- Companies are adopting big data for performing high-velocity real-time analytics on very large volumes of data to enable rapid analysis for business users using self-service and never-before-realized use cases. However, such projects have yielded limited value because these big data systems have become siloed from the rest of the enterprise systems holding critical business operational data. Big Data Fabric is a modern data architecture combining data virtualization, data prep, and lineage capabilities to seamlessly integrate at scale these huge, siloed volumes of structured and unstructured data with other enterprise data assets. This presentation will demonstrate with proven customer case studies in big data and IoT about the value of using big data fabric as a logical data lake for big data analytics.
Data Virtualization: From Zero to Hero (Middle East)Denodo
Watch full webinar here: https://bit.ly/3vPcmQ4
At the rate which enterprise data volume is increasing, replicating data to a central repository for analysis purposes is slow and expensive which might not even be a necessary part of the data integration process in many situations. With technologies such as data virtualization, companies can now place a single secure virtual layer between all disparate data sources (including both on-premise and in the cloud) on one side and various consuming applications on the other. Data replication for data integration is now an option and not a necessity.
Join us for this webinar to become a "Data Virtualization Hero" inside your organization.
In this session you will learn:
- What data virtualization really is
- How it differs from other enterprise data integration technologies
- Why data virtualization is finding enterprise-wide deployment inside some of the largest organizations
Multi-Cloud-Datenintegration mit DatenvirtualisierungDenodo
Watch full webinar here: https://bit.ly/3bcwEaS
Laut einer aktuellen Gartner Studie nutzen 81% der Organisationen Cloud-Services von zwei oder mehreren Anbietern, um eine hohe Flexibilität und bestmögliche Performance zu erzielen. Dies führt zu einer komplexen Infrastruktur, die sowohl das Auffinden von Daten als auch den Zugriff auf diese Daten erschwert.
Durch Datenvirtualisierung wird eine dezidierte Schicht für das Data-Discovery und den Datenzugriff angeboten. Durch eine „Multi-Location-Architektur“ wird den Nutzern ein umfänglicher und gemanagter Zugriff auf Daten ermöglicht. Dieser Zugriff ist unabhängig davon, ob sich die Daten in einem Rechenzentrum oder einer (beliebigen) Cloud befinden! Gleichzeitig behalten die "Data Owner" die lokale Kontrolle über ihre Daten und lokale Datenschutzbestimmungen (z.B. DSGVO) werden eingehalten.
Key Takeaways dieses Webinars:
- Herausforderungen bei der Einführung von Multi-Cloud-Datenstrategien
- Wie mit der Denodo-Plattform ein gemanagter „Data Access Layer“ für die gesamte Organisation bereitgestellt wird
- Multi-Location Architekturen, die gleichermaßen die lokale Kontrolle als auch den Echtzeitzugriff auf Daten ermöglichen
Watch full webinar here: https://bit.ly/2xc6IO0
To solve these challenges, according to Gartner "through 2022, 60% of all organizations will implement data virtualization as one key delivery style in their data integration architecture". It is clear that data virtualization has become a driving force for companies to implement agile, real-time and flexible enterprise data architecture.
In this session we will look at the data integration challenges solved by data virtualization, the main use cases and examine why this technology is growing so fastly. You will learn:
- What data virtualization really is
- How it differs from other enterprise data integration technologies
- Why data virtualization is finding enterprise-wide deployment inside some of the largest organizations
Denodo DataFest 2017: Business Needs for a Fast Data StrategyDenodo
1) ARGUS and Voyanta are now part of Altus Analytics. Voyanta provides data quality filtering and validation tools to help customers access and analyze data from multiple sources.
2) Voyanta's original solution involved large in-memory cube models for reporting and visualization, which had scaling and performance issues. The new architecture takes a modular approach with separate databases, data transformation, and visualization layers for improved reusability, consistency, and performance.
3) The "fast data" approach uses data virtualization to provide rapid access to transformed data and dashboards directly from raw sources. However, this can be slower for complex queries. Caching strategies like partial or full pre-building aim to balance fast access and up
The Virtualization of Clouds - The New Enterprise Data Architecture OpportunityDenodo
Watch full webinar here: https://bit.ly/3x7xVuR
Organizations worldwide are adopting a variety of the public cloud service providers (i.e. AWS, Google, Microsoft) and each have a portfolio of storage, compute, network, and security options. All of which create significant challenges in managing a hybrid and multi-cloud enterprise architecture. Even worse is the impact to the governance and integration of data from the clouds and physical infrastructure to support the broad array of analytics and operational requirements.
Can one public cloud provider meet all your needs today and in the future? How do you manage across multiple public and private clouds you have today and where your data exists? And, how would you manage and operate your multi-cloud and on-premises systems to gain value from your data in any of them? The Chief Research Officer at Ventana Research, Mark Smith, will expound the challenges and path ahead for virtualization and integration of your data and the clouds, setting an architectural path for best success.
Watch full webinar here: https://buff.ly/2mHGaLA
What started to evolve as the most agile and real-time enterprise data fabric, data virtualization is proving to go beyond its initial promise and is becoming one of the most important enterprise big data fabrics.
Attend this session to learn:
• What data virtualization really is
• How it differs from other enterprise data integration technologies
• Why data virtualization is finding enterprise-wide deployment inside some of the largest organizations
Maximizing Data Lake ROI with Data Virtualization: A Technical DemonstrationDenodo
Watch full webinar here: https://bit.ly/3ohtRqm
Companies with corporate data lakes also need a strategy for how to best integrate them with their overall data fabric. To take full advantage of a data lake, data architects must determine what data belongs in the Lake vs. other sources, how end users are going to find and connect to the data they need as well as the best way to leverage the processing power of the data lake. This webinar will provide you with a deep dive look at how the Denodo Platform for data virtualization enables companies to maximize their investment in their corporate data lake.
Watch on-demand this webinar to learn:
- How to create a logical data fabric with Denodo
- How to leverage the a data lake for MPP Acceleration and Summary Views
- How to leverage Presto with Denodo for file based data lakes (ie. S3, ADLS, HDFS, etc.)
The document outlines an upcoming Data Mesh Professionals Meetup Group meeting on January 28th. The meeting will include an overview of expectations, a keynote, experience sharing, and Q&A. The purpose is to deliberately share, learn, and explore data mesh principles and practices. The meeting is aimed at anyone who can influence, facilitate, implement, or operate analytical data and systems at scale, such as CIOs, CTOs, architects, and data scientists. A backlog of future meeting topics is also provided covering various technical and organizational aspects of data mesh.
According to Gartner, “By 2018, organizations with data virtualization capabilities will spend 40% less on building and managing data integration processes for connecting distributed data assets.” This solidifies Data Virtualization as a critical piece of technology for any flexible and agile modern data architecture.
This session will:
• Introduce data virtualization and explain how it differs from traditional data integration approaches
• Discuss key patterns and use cases of Data Virtualization
• Set the scene for subsequent sessions in the Packed Lunch Webinar Series, which will take a deeper dive into various challenges solved by data virtualization.
Agenda:
• Introduction & benefits of DV
• Summary & Next Steps
• Q&A
Watch full webinar here: https://goo.gl/EFQNFs
This webinar is part of the Data Virtualization Packed Lunch Webinar Series: https://goo.gl/W1BeCb
Data Mesh in Practice: How Europe’s Leading Online Platform for Fashion Goes ...Databricks
The Data Lake paradigm is often considered the scalable successor of the more curated Data Warehouse approach when it comes to democratization of data. However, many who went out to build a centralized Data Lake came out with a data swamp of unclear responsibilities, a lack of data ownership, and sub-par data availability.
Watch full webinar here: https://bit.ly/3dhbZTK
What started to evolve as the most agile and real-time enterprise data fabric, data virtualization is proving to go beyond its initial promise and is becoming one of the most important enterprise big data fabrics.
Watch this session to learn:
- What data virtualization really is.
- How it differs from other enterprise data integration technologies.
- Why data virtualization is finding enterprise-wide deployment inside some of the largest organizations.
Dr. Christian Kurze from Denodo, "Data Virtualization: Fulfilling the Promise...Dataconomy Media
This document discusses data virtualization and how it can help organizations leverage data lakes to access all their data from disparate sources through a single interface. It addresses how data virtualization can help avoid data swamps, prevent physical data lakes from becoming silos, and support use cases like IoT, operational data stores, and offloading. The document outlines the benefits of a logical data lake created through data virtualization and provides examples of common use cases.
Denodo DataFest 2017: Conquering the Edge with Data VirtualizationDenodo
Watch the live session on-demand: https://goo.gl/qAL3Q7
No time like the present! That's one reason why edge analytics continues to grow in value and importance. With the right analytic architecture in place, companies can not only identify opportunities at the edge, they can take appropriate actions.
Watch this Denodo DataFest 2017 session to discover:
• The growing importance of edge computing in IoT
• How data virtualization plays a critical role in enabling edge analytics
• How Denodo’s innovative customers exploit edge for a winning business model
Denodo DataFest 2016: The Role of Data Virtualization in IoT IntegrationDenodo
Watch the full session: Denodo DataFest 2016 sessions: https://goo.gl/DOrhiA
Connected use cases are gaining momentum! Data integration is the foundation for enabling these connections. In this session, you will experience first-hand our customer case studies and implementation architectures of IoT solutions.
In this session, you will learn:
• The role of data virtualization in enabling IoT use cases
• How our customers have successfully implemented IoT solutions using data virtualization
• How our product complements other IoT technologies
This session is part of the Denodo DataFest 2016 event. You can also watch more Denodo DataFest sessions on demand here: https://goo.gl/VXb6M6
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...HostedbyConfluent
Organizations have been chasing the dream of data democratization, unlocking and accessing data at scale to serve their customers and business, for over a half a century from early days of data warehousing. They have been trying to reach this dream through multiple generations of architectures, such as data warehouse and data lake, through a cambrian explosion of tools and a large amount of investments to build their next data platform. Despite the intention and the investments the results have been middling.
In this keynote, Zhamak shares her observations on the failure modes of a centralized paradigm of a data lake, and its predecessor data warehouse.
She introduces Data Mesh, a paradigm shift in big data management that draws from modern distributed architecture: considering domains as the first class concern, applying self-sovereignty to distribute the ownership of data, applying platform thinking to create self-serve data infrastructure, and treating data as a product.
This talk introduces the principles underpinning data mesh and Zhamak's recent learnings in creating a path to bring data mesh to life in your organization.
Augmentation, Collaboration, Governance: Defining the Future of Self-Service BIDenodo
Watch full webinar here: https://bit.ly/3zVJRRf
According to Dresner Advisory’s 2020 Self-Service Business Intelligence Market Study, 62% of the responding organizations say self-service BI is critical for their business. If we look deeper into the need for today’s self-service BI, it’s beyond some Executives and Business Users being enabled by IT for self-service dashboarding or report generation. Predictive analytics, self-service data preparation, collaborative data exploration are all different facets of new generation self-service BI. While democratization of data for self-service BI holds many benefits, strict data governance becomes increasingly important alongside.
In this session we will discuss:
- The latest trends and scopes of self-service BI
- The role of logical data fabric in self-service BI
- How Denodo enables self-service BI for a wide range of users - Customer case study on self-service BI
Limited Budget but Effective End to End MLOps Practices (Machine Learning Mod...IRJET Journal
This document proposes a low-cost approach to implementing a typical MLops pipeline for small organizations without expensive cloud platforms. It describes:
1. Using Python, R, SQL and shell scripts to manage the entire ML workflow on-premises, covering data management, model building/management, deployment, monitoring and continuous training.
2. Key elements like a centralized code/model repository, computation platform to execute code, and periodic deployment via cron jobs to integrate changes.
3. A model drift and continuous training process that retrains models if performance declines, and a data drift analysis method to measure parameter impacts.
4. Several use cases like customer churn modeling that can be effectively implemented this way
CD4ML and the challenges of testing and quality in ML systemsSeldon
Speaker: Danilo Sato, principal consultant at ThoughtWorks.
Bio: Danilo Sato (@dtsato) is a principal consultant at ThoughtWorks with experience in many areas of architecture and engineering: software, data, infrastructure, and machine learning. He is the author of "DevOps in Practice: Reliable and Automated Software Delivery", a member of ThoughtWorks Technology Advisory Board, and ThoughtWorks Office of the CTO.
Title: CD4ML and the challenges of testing and quality in ML systems
Abstract: Continuous Delivery for Machine Learning (CD4ML) deals with the challenges of applying Continuous Delivery principles to ML systems to make the end-to-end process of developing and deploying them more repeatable and reliable. These systems are generally more complex than traditional software applications, and ML models are non-deterministic and hard to explain. In this talk we will discuss the challenges of testing and quality in ML systems, and share some practices for applying different types of tests to help overcome those issues.
www.devopsinpractice.com
www.devopsnapratica.com.br
The Role of the Logical Data Fabric in a Unified Platform for Modern AnalyticsDenodo
Watch full webinar here: https://bit.ly/3FHKalT
Given the growing demand for analytics and the need for organizations to advance beyond dashboards to self-service analytics and more sophisticated algorithms like machine learning (ML), enterprises are moving towards a unified environment for data and analytics. What is the best approach to accomplish this unification?
In TDWI’s recent Best Practice Report, Unified Platforms for Modern Analytics, written by Fern Halper, TDWI VP Research, Senior Research Director for Advanced Analytics, adoption, use, challenges, architectures, and best practices for unified platforms for modern analytics is explored. One of the approaches for unification outlined in the report is a data fabric approach.
Join us for a webinar with our Director of Product Marketing, Robin Tandon, where he will discuss the role of the logical data fabric in a unified platform for modern analytics, focusing on several of the key findings outlined in this report. He will share insights and use case examples that demonstrate how a properly implemented logical data fabric is the most suitable approach for Unified Data Platforms across enterprises and organizations.
Watch on-demand & Learn:
- The benefits of a unified platform and its ability to capture diverse & emerging data types and how to support high performance and scalable solutions.
- The role of an enhanced AI driven data catalog and its implications towards the findings in the best practice report.
- Implications of a logical data fabric as it relates to several of the recommendations outlined in the report.
Minimizing the Complexities of Machine Learning with Data VirtualizationDenodo
Watch full webinar here: https://buff.ly/309CZ1Y
Advanced data science techniques, like machine learning, have proven an extremely useful tool to derive valuable insights from existing data. Platforms like Spark, and complex libraries for R, Python and Scala put advanced techniques at the fingertips of the data scientists. However, these data scientists spent most of their time looking for the right data and massaging it into a usable format. Data virtualization offers a new alternative to address these issues in a more efficient and agile way.
Attend this webinar and learn:
*How data virtualization can accelerate data acquisition and massaging, providing the data scientist with a powerful tool to complement their practice
*How popular tools from the data science ecosystem: Spark, Python, Zeppelin, Jupyter, etc. integrate with Denodo
*How you can use the Denodo Platform with large data volumes in an efficient way
*About the success McCormick has had as a result of seasoning the Machine Learning and Blockchain Landscape with data virtualization
Big Data Fabric for At-Scale Real-Time Analysis by Edwin RobbinsData Con LA
Abstract:- Companies are adopting big data for performing high-velocity real-time analytics on very large volumes of data to enable rapid analysis for business users using self-service and never-before-realized use cases. However, such projects have yielded limited value because these big data systems have become siloed from the rest of the enterprise systems holding critical business operational data. Big Data Fabric is a modern data architecture combining data virtualization, data prep, and lineage capabilities to seamlessly integrate at scale these huge, siloed volumes of structured and unstructured data with other enterprise data assets. This presentation will demonstrate with proven customer case studies in big data and IoT about the value of using big data fabric as a logical data lake for big data analytics.
Data Virtualization: From Zero to Hero (Middle East)Denodo
Watch full webinar here: https://bit.ly/3vPcmQ4
At the rate which enterprise data volume is increasing, replicating data to a central repository for analysis purposes is slow and expensive which might not even be a necessary part of the data integration process in many situations. With technologies such as data virtualization, companies can now place a single secure virtual layer between all disparate data sources (including both on-premise and in the cloud) on one side and various consuming applications on the other. Data replication for data integration is now an option and not a necessity.
Join us for this webinar to become a "Data Virtualization Hero" inside your organization.
In this session you will learn:
- What data virtualization really is
- How it differs from other enterprise data integration technologies
- Why data virtualization is finding enterprise-wide deployment inside some of the largest organizations
Multi-Cloud-Datenintegration mit DatenvirtualisierungDenodo
Watch full webinar here: https://bit.ly/3bcwEaS
Laut einer aktuellen Gartner Studie nutzen 81% der Organisationen Cloud-Services von zwei oder mehreren Anbietern, um eine hohe Flexibilität und bestmögliche Performance zu erzielen. Dies führt zu einer komplexen Infrastruktur, die sowohl das Auffinden von Daten als auch den Zugriff auf diese Daten erschwert.
Durch Datenvirtualisierung wird eine dezidierte Schicht für das Data-Discovery und den Datenzugriff angeboten. Durch eine „Multi-Location-Architektur“ wird den Nutzern ein umfänglicher und gemanagter Zugriff auf Daten ermöglicht. Dieser Zugriff ist unabhängig davon, ob sich die Daten in einem Rechenzentrum oder einer (beliebigen) Cloud befinden! Gleichzeitig behalten die "Data Owner" die lokale Kontrolle über ihre Daten und lokale Datenschutzbestimmungen (z.B. DSGVO) werden eingehalten.
Key Takeaways dieses Webinars:
- Herausforderungen bei der Einführung von Multi-Cloud-Datenstrategien
- Wie mit der Denodo-Plattform ein gemanagter „Data Access Layer“ für die gesamte Organisation bereitgestellt wird
- Multi-Location Architekturen, die gleichermaßen die lokale Kontrolle als auch den Echtzeitzugriff auf Daten ermöglichen
Watch full webinar here: https://bit.ly/2xc6IO0
To solve these challenges, according to Gartner "through 2022, 60% of all organizations will implement data virtualization as one key delivery style in their data integration architecture". It is clear that data virtualization has become a driving force for companies to implement agile, real-time and flexible enterprise data architecture.
In this session we will look at the data integration challenges solved by data virtualization, the main use cases and examine why this technology is growing so fastly. You will learn:
- What data virtualization really is
- How it differs from other enterprise data integration technologies
- Why data virtualization is finding enterprise-wide deployment inside some of the largest organizations
Denodo DataFest 2017: Business Needs for a Fast Data StrategyDenodo
1) ARGUS and Voyanta are now part of Altus Analytics. Voyanta provides data quality filtering and validation tools to help customers access and analyze data from multiple sources.
2) Voyanta's original solution involved large in-memory cube models for reporting and visualization, which had scaling and performance issues. The new architecture takes a modular approach with separate databases, data transformation, and visualization layers for improved reusability, consistency, and performance.
3) The "fast data" approach uses data virtualization to provide rapid access to transformed data and dashboards directly from raw sources. However, this can be slower for complex queries. Caching strategies like partial or full pre-building aim to balance fast access and up
The Virtualization of Clouds - The New Enterprise Data Architecture OpportunityDenodo
Watch full webinar here: https://bit.ly/3x7xVuR
Organizations worldwide are adopting a variety of the public cloud service providers (i.e. AWS, Google, Microsoft) and each have a portfolio of storage, compute, network, and security options. All of which create significant challenges in managing a hybrid and multi-cloud enterprise architecture. Even worse is the impact to the governance and integration of data from the clouds and physical infrastructure to support the broad array of analytics and operational requirements.
Can one public cloud provider meet all your needs today and in the future? How do you manage across multiple public and private clouds you have today and where your data exists? And, how would you manage and operate your multi-cloud and on-premises systems to gain value from your data in any of them? The Chief Research Officer at Ventana Research, Mark Smith, will expound the challenges and path ahead for virtualization and integration of your data and the clouds, setting an architectural path for best success.
Watch full webinar here: https://buff.ly/2mHGaLA
What started to evolve as the most agile and real-time enterprise data fabric, data virtualization is proving to go beyond its initial promise and is becoming one of the most important enterprise big data fabrics.
Attend this session to learn:
• What data virtualization really is
• How it differs from other enterprise data integration technologies
• Why data virtualization is finding enterprise-wide deployment inside some of the largest organizations
Maximizing Data Lake ROI with Data Virtualization: A Technical DemonstrationDenodo
Watch full webinar here: https://bit.ly/3ohtRqm
Companies with corporate data lakes also need a strategy for how to best integrate them with their overall data fabric. To take full advantage of a data lake, data architects must determine what data belongs in the Lake vs. other sources, how end users are going to find and connect to the data they need as well as the best way to leverage the processing power of the data lake. This webinar will provide you with a deep dive look at how the Denodo Platform for data virtualization enables companies to maximize their investment in their corporate data lake.
Watch on-demand this webinar to learn:
- How to create a logical data fabric with Denodo
- How to leverage the a data lake for MPP Acceleration and Summary Views
- How to leverage Presto with Denodo for file based data lakes (ie. S3, ADLS, HDFS, etc.)
The document outlines an upcoming Data Mesh Professionals Meetup Group meeting on January 28th. The meeting will include an overview of expectations, a keynote, experience sharing, and Q&A. The purpose is to deliberately share, learn, and explore data mesh principles and practices. The meeting is aimed at anyone who can influence, facilitate, implement, or operate analytical data and systems at scale, such as CIOs, CTOs, architects, and data scientists. A backlog of future meeting topics is also provided covering various technical and organizational aspects of data mesh.
According to Gartner, “By 2018, organizations with data virtualization capabilities will spend 40% less on building and managing data integration processes for connecting distributed data assets.” This solidifies Data Virtualization as a critical piece of technology for any flexible and agile modern data architecture.
This session will:
• Introduce data virtualization and explain how it differs from traditional data integration approaches
• Discuss key patterns and use cases of Data Virtualization
• Set the scene for subsequent sessions in the Packed Lunch Webinar Series, which will take a deeper dive into various challenges solved by data virtualization.
Agenda:
• Introduction & benefits of DV
• Summary & Next Steps
• Q&A
Watch full webinar here: https://goo.gl/EFQNFs
This webinar is part of the Data Virtualization Packed Lunch Webinar Series: https://goo.gl/W1BeCb
Data Mesh in Practice: How Europe’s Leading Online Platform for Fashion Goes ...Databricks
The Data Lake paradigm is often considered the scalable successor of the more curated Data Warehouse approach when it comes to democratization of data. However, many who went out to build a centralized Data Lake came out with a data swamp of unclear responsibilities, a lack of data ownership, and sub-par data availability.
Watch full webinar here: https://bit.ly/3dhbZTK
What started to evolve as the most agile and real-time enterprise data fabric, data virtualization is proving to go beyond its initial promise and is becoming one of the most important enterprise big data fabrics.
Watch this session to learn:
- What data virtualization really is.
- How it differs from other enterprise data integration technologies.
- Why data virtualization is finding enterprise-wide deployment inside some of the largest organizations.
Dr. Christian Kurze from Denodo, "Data Virtualization: Fulfilling the Promise...Dataconomy Media
This document discusses data virtualization and how it can help organizations leverage data lakes to access all their data from disparate sources through a single interface. It addresses how data virtualization can help avoid data swamps, prevent physical data lakes from becoming silos, and support use cases like IoT, operational data stores, and offloading. The document outlines the benefits of a logical data lake created through data virtualization and provides examples of common use cases.
Denodo DataFest 2017: Conquering the Edge with Data VirtualizationDenodo
Watch the live session on-demand: https://goo.gl/qAL3Q7
No time like the present! That's one reason why edge analytics continues to grow in value and importance. With the right analytic architecture in place, companies can not only identify opportunities at the edge, they can take appropriate actions.
Watch this Denodo DataFest 2017 session to discover:
• The growing importance of edge computing in IoT
• How data virtualization plays a critical role in enabling edge analytics
• How Denodo’s innovative customers exploit edge for a winning business model
Denodo DataFest 2016: The Role of Data Virtualization in IoT IntegrationDenodo
Watch the full session: Denodo DataFest 2016 sessions: https://goo.gl/DOrhiA
Connected use cases are gaining momentum! Data integration is the foundation for enabling these connections. In this session, you will experience first-hand our customer case studies and implementation architectures of IoT solutions.
In this session, you will learn:
• The role of data virtualization in enabling IoT use cases
• How our customers have successfully implemented IoT solutions using data virtualization
• How our product complements other IoT technologies
This session is part of the Denodo DataFest 2016 event. You can also watch more Denodo DataFest sessions on demand here: https://goo.gl/VXb6M6
How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...HostedbyConfluent
Organizations have been chasing the dream of data democratization, unlocking and accessing data at scale to serve their customers and business, for over a half a century from early days of data warehousing. They have been trying to reach this dream through multiple generations of architectures, such as data warehouse and data lake, through a cambrian explosion of tools and a large amount of investments to build their next data platform. Despite the intention and the investments the results have been middling.
In this keynote, Zhamak shares her observations on the failure modes of a centralized paradigm of a data lake, and its predecessor data warehouse.
She introduces Data Mesh, a paradigm shift in big data management that draws from modern distributed architecture: considering domains as the first class concern, applying self-sovereignty to distribute the ownership of data, applying platform thinking to create self-serve data infrastructure, and treating data as a product.
This talk introduces the principles underpinning data mesh and Zhamak's recent learnings in creating a path to bring data mesh to life in your organization.
Augmentation, Collaboration, Governance: Defining the Future of Self-Service BIDenodo
Watch full webinar here: https://bit.ly/3zVJRRf
According to Dresner Advisory’s 2020 Self-Service Business Intelligence Market Study, 62% of the responding organizations say self-service BI is critical for their business. If we look deeper into the need for today’s self-service BI, it’s beyond some Executives and Business Users being enabled by IT for self-service dashboarding or report generation. Predictive analytics, self-service data preparation, collaborative data exploration are all different facets of new generation self-service BI. While democratization of data for self-service BI holds many benefits, strict data governance becomes increasingly important alongside.
In this session we will discuss:
- The latest trends and scopes of self-service BI
- The role of logical data fabric in self-service BI
- How Denodo enables self-service BI for a wide range of users - Customer case study on self-service BI
Limited Budget but Effective End to End MLOps Practices (Machine Learning Mod...IRJET Journal
This document proposes a low-cost approach to implementing a typical MLops pipeline for small organizations without expensive cloud platforms. It describes:
1. Using Python, R, SQL and shell scripts to manage the entire ML workflow on-premises, covering data management, model building/management, deployment, monitoring and continuous training.
2. Key elements like a centralized code/model repository, computation platform to execute code, and periodic deployment via cron jobs to integrate changes.
3. A model drift and continuous training process that retrains models if performance declines, and a data drift analysis method to measure parameter impacts.
4. Several use cases like customer churn modeling that can be effectively implemented this way
CD4ML and the challenges of testing and quality in ML systemsSeldon
Speaker: Danilo Sato, principal consultant at ThoughtWorks.
Bio: Danilo Sato (@dtsato) is a principal consultant at ThoughtWorks with experience in many areas of architecture and engineering: software, data, infrastructure, and machine learning. He is the author of "DevOps in Practice: Reliable and Automated Software Delivery", a member of ThoughtWorks Technology Advisory Board, and ThoughtWorks Office of the CTO.
Title: CD4ML and the challenges of testing and quality in ML systems
Abstract: Continuous Delivery for Machine Learning (CD4ML) deals with the challenges of applying Continuous Delivery principles to ML systems to make the end-to-end process of developing and deploying them more repeatable and reliable. These systems are generally more complex than traditional software applications, and ML models are non-deterministic and hard to explain. In this talk we will discuss the challenges of testing and quality in ML systems, and share some practices for applying different types of tests to help overcome those issues.
www.devopsinpractice.com
www.devopsnapratica.com.br
près de 60% de leur temps à essayer de comprendre comment fonctionnent les applications sur lesquelles ils travaillent ?
CAST Imaging est un logiciel de Software Intelligence qui produit automatiquement la documentation/base de connaissances techniques de n’importe quelle application.
Celle-ci prend la forme de blueprints interactifs – dont les données sont stockées dans Neo4j – qui cartographient en détail tous les éléments d’une application et toutes leurs dépendances.
Peu importe qu’ils soient en train de développer, maintenir, moderniser ou simplement monter en compétence, les utilisateurs de CAST Imaging trouvent en quelques minutes les réponses aux questions qu’ils se posent sur le fonctionnement de l’application, sans passer des heures à fouiller dans le code !
Production machine learning: Managing models, workflows and risk at scaleAlex Housley
Explainability for machine learning (ML) algorithms is essential when moving to production, but it’s notoriously difficult to achieve transparency and navigate changing regulations, like the EU’s new AI legislation. In this session, Seldon CEO Alex Housley dives into what recent legislation means for organisations deploying ML, how they can protect their deployments with powerful monitoring capabilities and what to consider when building a robust ML deployment pipeline. With the support of real-world use cases, Alex also gives a practical guide to approaching deployment responsibly, and how to bake monitoring into ML pipelines to minimise risk and future-proof organisations.
Automated software modernisation is the best solution that is fast, low cost, preserves legacy value and is less risky by comparison to the traditional methodology of a re-write or replacement by packaged ERP. Object Management Groups (OMGs) Model Driven Architecture (MDA) methodology provides an automated model-driven reverse engineering and forward engineering process called Architecture Driven Modernisation (ADM) which has already been successfully adopted by a variety of high profile organisations such as Boeing, U.S. Air Force, Raytheon, EDS, Thales (European Aerospace) and numerous governments worldwide.
WORPCLOUD LTD is focused on being an Automated Software Modernisation Expert. We use OMG compliant tools and parsing techniques to extract all system information, business semantics and software artifacts into an XML repository called the Abstract Syntax Tree Metamodel. Next we use MDAs automated transformation procedures to generate new source code of your choice. Manual architecting of the target system are also performed before the transformation thus ensuring; speed, low cost and accuracy of the automated process combined with the flexibility & insight of human analysis.
Research reveals that application modernisation and migration budgets are currently very strong, covering between 25% to 71% of most companies IT budgets in 2013/2014. This clearly indicates that application modernisation is one of the most significant issues affecting companies – due to high software maintenance costs, low business flexibility and crippled integration and interoperability. Software modernisation is the sole remedy for these problems and your organisation can make huge savings by modernising.
Automated software modernisation is the best solution that is fast, low cost, preserves legacy value and is less risky by comparison to the traditional methodology of a re-write or replacement by packaged ERP. Object Management Groups (OMGs) Model Driven Architecture (MDA) methodology provides an automated model-driven reverse engineering and forward engineering process called Architecture Driven Modernisation (ADM) which has already been successfully adopted by a variety of high profile organisations such as Boeing, U.S. Air Force, Raytheon, EDS, Thales (European Aerospace) and numerous governments worldwide.
WORPCLOUD LTD is focused on being an Automated Software Modernisation Expert. We use OMG compliant tools and parsing techniques to extract all system information, business semantics and software artifacts into an XML repository called the Abstract Syntax Tree Metamodel. Next we use MDAs automated transformation procedures to generate new source code of your choice. Manual architecting of the target system are also performed before the transformation thus ensuring; speed, low cost and accuracy of the automated process combined with the flexibility & insight of human analysis.
Research reveals that application modernisation and migration budgets are currently very strong, covering between 25% to 71% of most companies IT budgets in 2013/2014. This clearly indicates that application modernisation is one of the most significant issues affecting companies – due to high software maintenance costs, low business flexibility and crippled integration and interoperability. Software modernisation is the sole remedy for these problems and your organisation can make huge savings by modernising.
ATMOSPHERE was invited to be a speaker at Think Milano event, on 6th June from 14.30 to 17.30, to join a panel discussion called “L’infrastruttura cloud ready protagonista del future” on how cloud infrastructures are important for different market sectors.
The REMICS model-driven process for migrating legacy applications to the cloudMarcos Almeida
With the advent of cloud computing platforms, many companies are studying the migration of legacy applications to the cloud. The main difficulty in dealing with such system is the obsolescence, either due to the dependency on an obsolete platform, incomplete/incorrect documentation or using an inappropriate architecture for the cloud. The FP7 project REMICS (Reuse and Migration of legacy applications to Interoperable Cloud Services) intends to provide a model-driven approach to extract valuable information from existing code and automating the refactoring of old code into cloud enabled architectures. In order to do so, REMICS proposes a process based on three steps: Recovery, Migration and Deployment. The work to be performed during each step is partially automated by the tools developed in the project. This presentation is going to focus on the description of the process and its associated tools and on our experience in applying the process in an industrial case study.
Extending open source and hybrid cloud to drive OT transformation - Future Oi...John Archer
A look at ESG concerns and agility needed to address pressures to transform energy organizations with decarbonization. Presented to Future Oil and Gas conference November 2021
Dagster - DataOps and MLOps for Machine Learning Engineers.pdfHong Ong
In this session, we will introduce Dagster, a cutting-edge framework that simplifies DataOps and MLOps for machine learning engineers. We will explore the benefits of this powerful tool, learn how to implement it in your machine learning workflows, and discuss practical use cases to help you enhance productivity, collaboration, and deployment of ML models.
¿Cómo modernizar una arquitectura de TI con la virtualización de datos?Denodo
Ver: https://bit.ly/347ImDf
En la era digital, la gestión eficiente de los datos es un factor fundamental para optimizar la competitividad de las empresas. Sin embargo, la mayoría de ellas se enfrentan a silos de datos, lo que hace que su tratamiento sea lento y costoso. Además, la velocidad, la diversidad y el volumen de los datos pueden superar las arquitecturas de TI tradicionales.
¿Cómo mejorar la entrega de datos para extraer todo su valor?
¿Cómo conseguir que los datos estén disponibles y poder utilizarlos en tiempo real?
Los expertos de Vault IT y Denodo te proponen este webinar para descubrir cómo la virtualización de datos permite modernizar una arquitectura de TI en un contexto de transformación digital.
The document discusses some of the promises and perils of mining software repositories like Git and GitHub for research purposes. It notes that while these sources contain rich data on software development, there are also challenges to consider. For example, decentralized version control systems like Git allow private collaboration that may be missed. And most GitHub projects are personal and inactive, while it is also used for storage and hosting. The document recommends researchers approach these data sources carefully and provides lessons on how to properly analyze and interpret the data from repositories like Git and GitHub.
From Model-based to Model and Simulation-based Systems ArchitecturesObeo
Achieving quality engineering through descriptive and analytical models
Systems architecture design is a key activity that affect the
overall systems engineering cost. It is hence fundamental
to ensure that the system architecture reaches a proper quality.
In this paper, we leverage on MBSE approaches and complement them
with simulation techniques, as a prom-ising way to improve the quality of the system architecture definition, and to come up with inno-vative solutions while securing the systems engineering process.
Data Science as a Service: Intersection of Cloud Computing and Data SciencePouria Amirian
Dr. Pouria Amirian explains data science, steps in a data science workflow and show some experiments in AzureML. He also mentions about big data issues in a data science project and solutions to them.
Data Science as a Service: Intersection of Cloud Computing and Data SciencePouria Amirian
Dr. Pouria Amirian from the University of Oxford explains Data Science and its relationship with Big Data and Cloud Computing. Then he illustrates using AzureML to perform a simple data science analytics.
This document discusses the challenges of heterogeneous data models when dealing with multi-cloud and multi-data store applications. Specifically, it notes that with different cloud providers come different data stores and data models, leading to fragmentation. The document proposes that model-driven engineering can help address this issue by separating the problem from the solution, and enabling design data to be stored anywhere through transformation. However, writing transformations is challenging due to differences in data store paradigms and variable data structures across targets.
Notes on Deploying Machine-learning Models at ScaleDeep Kayal
While modeling techniques in machine learning have matured drastically, the deployment of models at scale has been overlooked. These are some learnings that I've had over the years, that I presented at Cognizant in Amsterdam.
This document discusses engineering digitalization through task automation and reuse in the development lifecycle. It proposes a knowledge-centric approach to systems engineering using a knowledge management strategy. This includes defining a controlled vocabulary, relating terms through relationships and clusters, representing textual patterns for matching, and combining rules and tasks to infer information. This knowledge graph could then enable capabilities like requirements extraction, model population, quality checking, and reuse of system artifacts. The approach aims to automate tasks, link different artifact types, and leverage semantics and AI/ML to better understand and exploit knowledge embedded in systems artifacts.
[DSC Europe 22] Reproducibility and Versioning of ML Systems - Spela PoklukarDataScienceConferenc1
Reproducibility of ML system is increasingly important topic in ML community. Reproducibility ensures conclusiveness of the model performance, provides an understanding how ML system works and reduces unnecessary errors when the system is deployed into production. With increasing AI regulation, it will soon become a requirement for many ML applications. In this talk, we will explore different aspects of reproducibility such as reproducibility of the dataset, data processing, ML model, its randomness and hyperparameters, code and SW environment, as well as concepts and practical tools such as data versioning, feature, metadata and artifact store, model registry and containerization that together ensure reproducibility of our experiments.
Similar to Continuous Intelligence: Keeping your AI Application in Production (20)
Continuous Intelligence: Keeping Your AI Application in Production (NDC Sydne...Dr. Arif Wider
A talk about applying Continuous Delivery to Machine Learning (CD4ML) presented by Arif Wider from ThoughtWorks at NDC Sydney Conference 2019.
Abstract:
It is already challenging to transition a machine learning model or AI system from the research space to production, and maintaining that system alongside ever-changing data is an even greater challenge. In software engineering, Continuous Delivery practices have been developed to ensure that developers can adapt, maintain, and update software and systems cheaply and quickly, enabling release cycles on the scale of hours or days instead of weeks or months. Nevertheless, in the data science world Continuous Delivery is rarely been applied holistically.
This is partly due to different workflows: data scientists regularly work on whole sets of hypotheses, whereas software engineers work more linearly even when evaluating multiple implementation alternatives. Therefore, existing software engineering practices cannot be applied as-is to machine learning projects. Learn how we used our expertise in both fields to adapt practices and tools to allow for Continuous Intelligence–the practice of delivering AI applications continuously.
Continuous Intelligence: Moving Machine Learning into Production ReliablyDr. Arif Wider
A workshop by Danilo Sato, Christoph Windheuser, Emily Gorcenski, and Arif Wider, given at Strata Data Conference 2019 in London.
Abstract:
So you want to include a machine learning component in your IT systems? The process is a little more involved than clicking through an AI tutorial on your laptop. It’s not just the first working model you run that you need to consider; you also need to think about things like integration, scaling, and testing. What’s more, postlaunch, you’ll want to continuously adapt your model to respond to the changing environment.
ThoughtWorks pioneered continuous delivery—a set of tools and processes that ensure that software under development can be reliably released to production at any time and with high frequency.
Danilo Sato and Christoph Windheuser demonstrate how to apply continuous delivery to machine learning—what’s known as continuous intelligence. In a live scenario, you’ll change a machine learning model in a development environment, test its new performance, and, depending on the outcome, automatically deploy the new model into a production environment. The tech stack for this scenario will be Python, DVC (Data Science Version Control), and GoCD.
Continuous Intelligence: Keeping your AI Application in ProductionDr. Arif Wider
A talk by Emily Gorcenski and Arif Wider presented a Strata Data Conference 2019 in London.
Abstract:
It’s already challenging to transition a machine learning model or AI system from the research space to production, and maintaining that system alongside ever-changing data is an even greater challenge. In software engineering, continuous delivery practices have been developed to ensure that developers can adapt, maintain, and update software and systems cheaply and quickly, enabling release cycles on the scale of hours or days instead of weeks or months.
Nevertheless, in the data science world, continuous delivery is rarely applied holistically—due in part to different workflows: data scientists regularly work on whole sets of hypotheses, whereas software engineers work more linearly even when evaluating multiple implementation alternatives. Therefore, existing software engineering practices cannot be applied as is to machine learning projects.
Arif Wider and Emily Gorcenski explore continuous delivery (CD) for AI/ML along with case studies for applying CD principles to data science workflows. Join in to learn how they drew on their expertise to adapt practices and tools to allow for continuous intelligence—the practice of delivering AI applications continuously.
DataDevOps: A Manifesto for a DevOps-like Culture Shift in Data & AnalyticsDr. Arif Wider
A talk by Sebastian Herold & Dr. Arif Wider at TDWI 2018 Munich.
Abstract:
More and more companies migrate their monolithic applications to a microservices architecture. However, maintaining a consistent and usable data landscape has only become more challenging by this: huge amounts of structured and unstructured data, and hundreds of data sources.
Furthermore, data-driven product development multiplies the analytics requirements: every product team needs constantly updated and specially tailored metrics which often combine product specific data with company wide data.
Having a centralized data team does not scale in this setting as it becomes the bottleneck between data producers and data consumers.
We created a Manifesto based on five general themes which break with traditional separation of roles and show a path how to deal with distributed data in a federal and scalable fashion. This leads to DataDev: a culture shift similar to DevOps in which application developers own their data and take over responsibilities for data & analytics.
Learn about our experiences and best practices with facilitating this cultural transformation at Zalando, one of Europe's largest online fashion platforms.
DataDevOps: A Manifesto for a DevOps-like Culture Shift in Data & AnalyticsDr. Arif Wider
A talk given by Dr. Arif Wider (ThoughtWorks) and Sebastian Herold (Zalando) at OOP 2018 in Munich.
Abstract:
More and more companies migrate their monolithic applications to a microservices architecture. However, maintaining a consistent and usable data landscape has only become more challenging by this: huge amounts of structured and unstructured data, and hundreds of data sources.
Furthermore, data-driven product development multiplies the analytics requirements: every product team needs constantly updated and specially tailored metrics which often combine product specific data with company wide data.
Having a centralized data team does not scale in this setting as it becomes the bottleneck between data producers and data consumers.
We created a Manifesto of seven principles which break with traditional separation of roles and show a path how to deal with distributed data in a federal and scalable fashion. This leads to DataDev: a culture shift similar to DevOps in which application developers own their data and take over responsibilities for data & analytics.
Learn about our experiences and best practices with facilitating this cultural transformation at Scout24, the provider of Europe’s largest online markets for cars and real estate.
DataDevOps - A Manifesto on Shared Data Responsibility in Times of MicroservicesDr. Arif Wider
A talk by Sebastian Herold (Scout24) and Arif Wider (ThoughtWorks)
Abstract:
More and more companies successfully migrate their monolithic applications to a Microservices architecture. However, maintaining a consistent and usable data landscape has only become more challenging by this: unstructured data, huge amounts of data, and hundreds of data sources. Having a centralized data team does not scale in this setting as it becomes the bottleneck between application developers and business analysts.
We created a Data Manifesto of seven principles which break with traditional role separations and show a path how to deal with distributed data in a federal and scalable fashion. This leads to DataDevOps: a culture where application developers also own their data. Learn about the experiences we made with facilitating this cultural transformation at Scout24, the provider of Europe’s largest online markets for cars and real estate.
Predictive Analytics for Vehicle Price Prediction - Delivered Continuously at...Dr. Arif Wider
This document discusses predictive analytics for vehicle price prediction delivered continuously at AutoScout24. It describes AutoScout24's use of a random forest model for price prediction and their approach to automatically generating Java code from the R-based model to deploy it as a high-performance web application via a continuous delivery pipeline. Key lessons learned include forming cross-functional data science and engineering teams, setting up early usage reporting to improve the product, and addressing challenges of generating large amounts of Java code like optimizing garbage collection.
A High-Performance Solution to Microservice UI Composition @ XConf HamburgDr. Arif Wider
This document proposes an approach called Jigsaw for composing microservice user interfaces that balances team autonomy and page performance. Jigsaw defines pages as publicly accessible endpoints that include fragments from other services using server-side includes. The fragments adhere to contracts and can be cached. An Nginx proxy handles routing to services and uses modules like ngx_pagespeed to optimize and combine assets for improved page load performance when composing the user interface from multiple autonomous microservices.
An Unexpected Solution to Microservices UI CompositionDr. Arif Wider
This document describes a company's transition from a monolithic .NET application to a microservices architecture hosted on AWS. It details their use of a UI composition pattern using Nginx, ESI, and Jigsaw to compose fragments from different services into complete pages. This allows independent deployment of features while maintaining performance. Key aspects include caching of assets and responses, combining stylesheets and scripts, and isolation of services through separate CSS and JS packages.
How to Optimize Call Monitoring: Automate QA and Elevate Customer ExperienceAggregage
The traditional method of manual call monitoring is no longer cutting it in today's fast-paced call center environment. Join this webinar where industry experts Angie Kronlage and April Wiita from Working Solutions will explore the power of automation to revolutionize outdated call review processes!
The "Zen" of Python Exemplars - OTel Community DayPaige Cruz
The Zen of Python states "There should be one-- and preferably only one --obvious way to do it." OpenTelemetry is the obvious choice for traces but bad news for Pythonistas when it comes to metrics because both Prometheus and OpenTelemetry offer compelling choices. Let's look at all of the ways you can tie metrics and traces together with exemplars whether you're working with OTel metrics, Prom metrics, Prom-turned-OTel metrics, or OTel-turned-Prom metrics!
Guidelines for Effective Data VisualizationUmmeSalmaM1
This PPT discuss about importance and need of data visualization, and its scope. Also sharing strong tips related to data visualization that helps to communicate the visual information effectively.
Communications Mining Series - Zero to Hero - Session 2DianaGray10
This session is focused on setting up Project, Train Model and Refine Model in Communication Mining platform. We will understand data ingestion, various phases of Model training and best practices.
• Administration
• Manage Sources and Dataset
• Taxonomy
• Model Training
• Refining Models and using Validation
• Best practices
• Q/A
TrustArc Webinar - Your Guide for Smooth Cross-Border Data Transfers and Glob...TrustArc
Global data transfers can be tricky due to different regulations and individual protections in each country. Sharing data with vendors has become such a normal part of business operations that some may not even realize they’re conducting a cross-border data transfer!
The Global CBPR Forum launched the new Global Cross-Border Privacy Rules framework in May 2024 to ensure that privacy compliance and regulatory differences across participating jurisdictions do not block a business's ability to deliver its products and services worldwide.
To benefit consumers and businesses, Global CBPRs promote trust and accountability while moving toward a future where consumer privacy is honored and data can be transferred responsibly across borders.
This webinar will review:
- What is a data transfer and its related risks
- How to manage and mitigate your data transfer risks
- How do different data transfer mechanisms like the EU-US DPF and Global CBPR benefit your business globally
- Globally what are the cross-border data transfer regulations and guidelines
DynamoDB to ScyllaDB: Technical Comparison and the Path to SuccessScyllaDB
What can you expect when migrating from DynamoDB to ScyllaDB? This session provides a jumpstart based on what we’ve learned from working with your peers across hundreds of use cases. Discover how ScyllaDB’s architecture, capabilities, and performance compares to DynamoDB’s. Then, hear about your DynamoDB to ScyllaDB migration options and practical strategies for success, including our top do’s and don’ts.
Radically Outperforming DynamoDB @ Digital Turbine with SADA and Google CloudScyllaDB
Digital Turbine, the Leading Mobile Growth & Monetization Platform, did the analysis and made the leap from DynamoDB to ScyllaDB Cloud on GCP. Suffice it to say, they stuck the landing. We'll introduce Joseph Shorter, VP, Platform Architecture at DT, who lead the charge for change and can speak first-hand to the performance, reliability, and cost benefits of this move. Miles Ward, CTO @ SADA will help explore what this move looks like behind the scenes, in the Scylla Cloud SaaS platform. We'll walk you through before and after, and what it took to get there (easier than you'd guess I bet!).
The document discusses fundamentals of software testing including definitions of testing, why testing is necessary, seven testing principles, and the test process. It describes the test process as consisting of test planning, monitoring and control, analysis, design, implementation, execution, and completion. It also outlines the typical work products created during each phase of the test process.
The Strategy Behind ReversingLabs’ Massive Key-Value MigrationScyllaDB
ReversingLabs recently completed the largest migration in their history: migrating more than 300 TB of data, more than 400 services, and data models from their internally-developed key-value database to ScyllaDB seamlessly, and with ZERO downtime. Services using multiple tables — reading, writing, and deleting data, and even using transactions — needed to go through a fast and seamless switch. So how did they pull it off? Martina shares their strategy, including service migration, data modeling changes, the actual data migration, and how they addressed distributed locking.
CNSCon 2024 Lightning Talk: Don’t Make Me Impersonate My IdentityCynthia Thomas
Identities are a crucial part of running workloads on Kubernetes. How do you ensure Pods can securely access Cloud resources? In this lightning talk, you will learn how large Cloud providers work together to share Identity Provider responsibilities in order to federate identities in multi-cloud environments.
Test Management as Chapter 5 of ISTQB Foundation. Topics covered are Test Organization, Test Planning and Estimation, Test Monitoring and Control, Test Execution Schedule, Test Strategy, Risk Management, Defect Management
Introducing BoxLang : A new JVM language for productivity and modularity!Ortus Solutions, Corp
Just like life, our code must adapt to the ever changing world we live in. From one day coding for the web, to the next for our tablets or APIs or for running serverless applications. Multi-runtime development is the future of coding, the future is to be dynamic. Let us introduce you to BoxLang.
Dynamic. Modular. Productive.
BoxLang redefines development with its dynamic nature, empowering developers to craft expressive and functional code effortlessly. Its modular architecture prioritizes flexibility, allowing for seamless integration into existing ecosystems.
Interoperability at its Core
With 100% interoperability with Java, BoxLang seamlessly bridges the gap between traditional and modern development paradigms, unlocking new possibilities for innovation and collaboration.
Multi-Runtime
From the tiny 2m operating system binary to running on our pure Java web server, CommandBox, Jakarta EE, AWS Lambda, Microsoft Functions, Web Assembly, Android and more. BoxLang has been designed to enhance and adapt according to it's runnable runtime.
The Fusion of Modernity and Tradition
Experience the fusion of modern features inspired by CFML, Node, Ruby, Kotlin, Java, and Clojure, combined with the familiarity of Java bytecode compilation, making BoxLang a language of choice for forward-thinking developers.
Empowering Transition with Transpiler Support
Transitioning from CFML to BoxLang is seamless with our JIT transpiler, facilitating smooth migration and preserving existing code investments.
Unlocking Creativity with IDE Tools
Unleash your creativity with powerful IDE tools tailored for BoxLang, providing an intuitive development experience and streamlining your workflow. Join us as we embark on a journey to redefine JVM development. Welcome to the era of BoxLang.
2. 7000+ technologists with 43 offices in 14 countries
Partner for technology driven business transformation
Barcelona - Madrid - London - Manchester - Berlin - Hamburg - Munich - Cologne
15. NEW CHALLENGES
15
→ How to version control training data?
→ Training data and prediction models don’t fit into Git :-(
→ Model re-training slows down the entire continuous delivery server!
→ Data scientists want to evaluate several solutions at the same time...
→ ...and they use analytics notebooks which are hard to version control!
→ How to unit test data science code that is tied to changing data?
→ How to prevent behaviour changes of the model to break the application?
17. Continuous Delivery is the ability to get
changes of all types — including new features,
configuration changes, bug fixes and
experiments — into production, or into the
hands of users, safely and quickly in a
sustainable way.
Jez Humble & Dave Farley
雅虎竞拍
煤炉代买
paypay商城
乐天二手
日本亚马逊
乐天新品
ZOZOTOWN
