Data Mesh using Microsoft Fabric

Editor's Notes

1. Before we dive deeper, I want to run very quickly thru some basic assumptions which frames any architecture. The first assumption is that every application which processes data, needs to have some type of data persistency. Second, I note that applications are used to solve specific problems. Applications are unique and so it the data. This is because there are several stages to the design and development of applications. You always start with conceptual thinking and design; then you translate our knowledge to a logical application data model, which is an abstract structure of conceptual information and requirements. Finally, you make the physical application data model: the true design of the application and database. The physical data model is unique and receives both the context and nonfunctional requirements for how the application and database will be designed and used.
2. And these unique designs lead to another problem from which we can’t escape. It’s the data integration that is always around the corner when moving data across applications. There’s no escape from this dilemma, and it doesn’t matter you do ETL, ELT, virtual of physical, batch or real-time. This problem is always there. Any architecture is framed by these objectives.
3. As an architect I can tell you that the world heading towards distributed data at large. Several trends are fragmenting the data landscape, of which some you see on the screen. The first trend I see is an explosion of analytical tools and ways in which you can process and use your data. The consequence of this is that the same data ends up everywhere. A second trend is the Cloud, Services and API connectivity which pushes the data usage and distribution even further. At the same time, we need to be very much in control of our data, because of stronger regulation such as GDPR and BCBS. Next, I see a trend of increased compute power, which allows us to quickly move data across platforms and different locations. These trends of data distribution at scale will also exponentially grow data even further. And lastly, I see a trend where the read vs write ratio changes. Transactional systems we no longer use for only store and processing data for transactional purposes. They at the same time need to serve out spontaneously tons of data, which at the same time can be challenging.
4. Zhamak DDD and it’s org aspects Domain Zones and how they are independent and enabled Within a Domain Zone you create Data Products Which then can be consumed by other domains Creating a Data Mesh
5. empathized with today’s pain points of architectural and organizational challenges in order to become data-driven, use data to compete, or use data at scale to drive value. 
6. The data management landing zone has a management function and Is responsible for the governance of your analytics platform. The data management landing zone is responsible for the following: Data catalog Data quality management Data security and privacy Data governance
7. Zoom in on one domain zone In a data mesh, a domain zone is a way to define boundaries around your enterprise data. Domains can vary depending on your organization, and in some cases, you might want to define domains based on your line of business (LOB) According to Microsoft’s Cloud Adoption Framework, here are some best practices to follow: Use automation to create domain zones and ensure that they are consistent across your organization. Implement data services that are specific to each domain zone. Build data products that are specific to each domain zone. Share data products across domain zones to promote reuse and collaboration
8. Microsoft Purview provides a unified data governance solution to help manage and govern your on-premises, multicloud, and software as a service (SaaS) data. Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, real-time analytics, and business intelligence. It provides a way to organize data into domains. Domains are a way to logically group together all the data in an organization that is relevant to a particular area or field . OneLake is a single, unified, logical data lake for your whole organization - OneLake brings customers one data lake for the entire organization - one copy of data for use with multiple analytical engines - ability to organize and manage data in a logical way allowing different business groups to efficiently operate and control their own data.
9. Challenges of lego block architecture – too complex Clients, partners and every cloud provider is pushing to build an end to end data and analytics ecosystem using the “lego block” approach The approach is too complicated to implement at scale requirement different skills to ensure proper design and deployment (integration, security, networking, governance, etc) The challenge MS is solving is how do we simplify this implementation, how do we make it easier for our clients while ensuring all the enterprise requirements are met. Microsoft answer is the Analytics Continuum – our strategy and vision we’re executing on. Every standalone component of this architecture has six enterprise needs which must be met. In the architecture shown, including the cloud service, that could mean 36 points of failure, inefficiency or cost. Every additional cloud platform also incurs its own burden.
10. - Data domain & data mesh - Enterprise Scale for Analytics (Data Management and Analytics Scenario) - Microsoft framework available on public documentation - Guidance, best practices, deployment templates
11. MS Fabric gives the flexibility in each domain to build their own data products. Bottom to top: operational data sources: e.g. cosmos db -> fabric mirror, makes it accessable in the data hub -> can make new data products -> can be used in other domains Microsoft’s hybrid approach to data mesh, data fabric and data hub is based on the idea of combining the best features of each concept to create a data platform that is decentralized, scalable, and accessible. Microsoft Fabric’s data mesh architecture supports the data mesh principle by allowing data to be grouped into domains based on different business areas, such as marketing, sales, human resources, etc. Each domain has its own data owners, contributors, and governance rules, enabling decentralized data management and autonomy.  Microsoft Fabric also provides a OneLake data hub that makes it easy to find, explore, and use the data items in the organization that the user has access to. The data hub provides a filterable list of all the data items, a gallery of recommended data items, a way of finding data items by workspace or domain, and an options menu of things the user can do with the data item. The data hub also integrates with various data sources and services, such as Azure Synapse Analytics, Azure Data Factory, Azure Purview, and Power BI, to enable data ingestion, transformation, analysis, and visualization. The hybrid approach also leverages the data fabric technology to enable data integration, orchestration, and processing across different data sources and platforms. The hybrid data fabric and data mesh framework can help organizations design a data platform that can handle complex data scenarios, such as data streaming, data lake, data warehouse, data virtualization, data catalog, and data governance. The hybrid framework can also support various data products that can benefit from both data fabric technology and data mesh principles, such as data quality, data lineage, data security, data privacy, and data discovery. The hybrid approach aims to create a data platform that is flexible, agile, and adaptable to the changing data needs and requirements of the organization.
12. MS Fabric gives the flexibility in each domain to build their own data products. Bottom to top: operational data sources: e.g. cosmos db -> fabric mirror, makes it accessable in the data hub -> can make new data products -> can be used in other domains Microsoft’s hybrid approach to data mesh, data fabric and data hub is based on the idea of combining the best features of each concept to create a data platform that is decentralized, scalable, and accessible.  Microsoft Fabric’s data mesh architecture supports the data mesh principle by allowing data to be grouped into domains based on different business areas, such as marketing, sales, human resources, etc. Each domain has its own data owners, contributors, and governance rules, enabling decentralized data management and autonomy.  Microsoft Fabric also provides a OneLake data hub that makes it easy to find, explore, and use the data items in the organization that the user has access to. The data hub provides a filterable list of all the data items, a gallery of recommended data items, a way of finding data items by workspace or domain, and an options menu of things the user can do with the data item.. The hybrid approach also leverages the data fabric technology to enable data integration, orchestration, and processing across different data sources and platforms. The hybrid data fabric and data mesh framework can help organizations design a data platform that can handle complex data scenarios, such as data streaming, data lake, data warehouse, data virtualization, data catalog, and data governance. The hybrid framework can also support various data products that can benefit from both data fabric technology and data mesh principles, such as data quality, data lineage, data security, data privacy, and data discovery. The hybrid approach aims to create a data platform that is flexible, agile, and adaptable to the changing data needs and requirements of the organization.
13. The open data Lakehouse can be used as the technical foundation for data mesh. Data mesh aims to enable domains (often manifesting as business units in an enterprise) to use best-of-breed technologies to support their use cases
14. One security uses a layered security model built around the organizational structure of experiences within Microsoft Fabric, such as OneLake, Warehouse, Real-time analytics, and Power BI semantic models. One security allows you to manage security at different levels, such as workspace, item, and compute-specific security.
15. Domains are an integral part of Fabric. They are defined with domain admins and contributors who can associate workspaces and group them together under a relevant domain. Federated governance can be achieved by delegating settings to domain admins, thus allowing them to achieve more granular control over their business area. Domains simplify discovery and consumption of data across the organization, thus allowing business optimized consumption.
16. Tenant -> Domain -> Workspace Different business groups are now able to work independently within the same data lake without the overhead of managing different storage resources. They are already able to implement the popular data mesh pattern more efficiently than they could before. OneLake takes this even further with the introduction of domains as a first-class concept. A single business domain may have multiple workspaces as workspaces tend to align with specific projects or teams. A domain is a way of logically grouping together all the data in and organization that is relevant to an area or field. Domains are defined with domain admins and contributors who can logically group together workspaces under those domains.. Domains provide a management boundary between tenant and workspace enabling admins to have more granular control over multiple workspaces. As you will see later, domains also simplify discovery and consumption of data across the entire organization. Now that we are making it so easy for different parts of the organization to work on the same data lake without going through a central gatekeeper, you might be thinking that you want to block certain users from adding to the lake. If anyone can add to the data lake, then these can quickly become data swamps with data from official sources mixed with data from unofficial sources. The problem with blocking users from OneLake, is that they will just create another data lake somewhere else. When they do that, you will have no idea if that data is properly governed or even how it is being used. If they add their data to OneLake, it will be automatically governed and still under the control of the admins who will start to get more and more insights on how that data is being used. You can avoid data swamps in OneLake through data endorsements. Domain owners can officially certify data or recommend data so that the important data rises to the surface while the rest sinks to the bottom.
17. Think of OneLake as an abstraction layer. You can mount existing ADLS Gen2 to it. Virtualization across many storage account. Maintains a single namespace. A shortcut is nothing more than a symbolic link which points from one data location to another. Just like you can create shortcuts in Windows or Linux, the data will appear in the shortcut location as if it were physically there. Today, if you have tables in a data warehouse, which you want you want to make available along side other tables or files in a lakehouse, you will need to copy that data out of warehouse. With OneLake, you simply create a shortcut in the lakehouse pointing to the warehouse. The data will appear in your lakehouse as if you had physically copied it. Since you didn’t copy it, when data is updated in the warehouse, changes are automatically reflected in the lakehouse. You can also use shortcuts to consolidate data across workspace and domains without changing the ownership of the data. In this example, the workspace B still owns the data. They still have ultimate control over who can access it and how it stays up to date. Many of you already have existing data lakes stored in ADLS gen2 or in Amazon S3 buckets. These lakes can continue to exist and be managed externally to Fabric. We have extended shortcuts to include lake outside of OneLake and even outside of Azure so that you can virtualize you existing ADLS gen 2 accounts or Amazon S3 buckets into OneLake. All data is mapped to the same unified namespace and can be accessed using the same ADLS gen2 APIs even when it is coming from S3.
18. If we zoom out, we can see all these domains in OneLake. To get 360-degree view of your business, a single data item, or product in data mesh terms, will need to span multiple domains. It is shortcuts that provide the connections between domains so that data can be virtualized into a single data product without data movement, data duplication or changing the ownership of the data.
19. Sure, I can write you a demo script for Microsoft Fabric showcasing Data Mesh. Here is a possible example: Hello and welcome to this demo of Microsoft Fabric, the AI-powered analytics platform that helps you bring your data into the era of AI. In this demo, we will show you how Fabric enables you to leverage the power of Data Mesh, a decentralized data architecture that organizes data by business domains and provides more ownership to the data producers. Data Mesh is a concept that was introduced by Zhamak Dehghani in 20191 and is based on four principles: domain-oriented decentralized data ownership and architecture, data as a product, self-serve data infrastructure as a platform, and federated computational governance23. These principles aim to address the challenges of centralized, monolithic data structures, such as data accessibility, quality, and organization. With Fabric, you can implement Data Mesh in your organization by following these steps: Identify your business domains and the data producers and consumers for each domain. For example, you may have domains such as marketing, sales, customer service, and finance, each with their own data sources, pipelines, and analytics needs. Empower your domain teams to take responsibility for their data and treat it as a product. This means that the domain teams should design, build, and run their own data platforms, APIs, and services, using the Fabric tools and services that suit their needs. For example, they can use OneLake to create and manage their data lakes, Synapse to perform data engineering and data science, Power BI to create and share dashboards and reports, and Data Factory to orchestrate data movement and transformation. Enable self-service data access and discovery across domains by using Fabric’s data catalog and metadata management features. This allows the domain teams to document and expose their data products to other domains, as well as to consume data products from other domains, using standard protocols and formats. For example, they can use Data Activator to automatically generate insights and trigger actions from their data, or use Data Explorer to search and browse data products from different domains. Establish federated governance and compliance policies for your data mesh by using Fabric’s data security and quality features. This ensures that the data products are reliable, consistent, and trustworthy, and that the data consumers have the appropriate permissions and usage rights. For example, they can use Data Protector to monitor and protect their data from threats and breaches, or use Data Auditor to audit and validate their data quality and lineage. By following these steps, you can create a data mesh architecture that leverages the benefits of Fabric’s unified data foundation, role-tailored tools, AI-powered capabilities, and open, governed foundation. With Fabric and Data Mesh, you can reshape how your entire team uses data and drive innovation and growth for your business. Thank you for watching this demo of Microsoft Fabric and Data Mesh. If you want to learn more, please visit our website4 or sign up for a free trial5.