ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake

Editor's Notes

1. Alright. How many of you have dealt with compliance before? Either through enforcing it Or through adhering to compliance rules Okay, looks like almost all of you.
2. So, more likely than not, you have had days where you have felt like our friend. the staff engineer here. who's very overwhelmed with terms that he keeps hearing, such as GDPR, CCPA, DMA, PII, right to be forgotten, privacy by design, and so many of those buzzwords. On the other side, he's responsible for managing a very large data lake with so many data applications on top of it and wants to make everything work. This sounds like a very challenging problem.
3. At the same time, privacy by default is on the rise. Companies managing user data are implementing various controls to empower users to manage their privacy and security effectively. They provide tools such as privacy dashboards and options to export user data , which might be used in other services or retained for records. There are also tools to adjust ad preferences to control what type of data the platform can use to display ads to users, alongside frequent security checkups and options to delete personal data from the site.
4. Although managing compliance might sound overwhelming, the solution is surprisingly simple. To handle compliance, you only need two key components: one is the policy engine, and the other is the query engine.
5. Here’s how it works: Initially, policies inferred from regulations or internal guidelines are represented in a structured format and are kept in a policy store. This is fed into the policy engine along with data lake metadata, which includes table schemas along with column policy annotations. The policy engine then produces a set of SQL views encoding the necessary transformations according to data usage. These views are subsequently fed into a query engine along with the data and user applications, and are used to implement compliant data applications. Therefore, this workflow simplifies the journey from a complex maze of policies and tables in the data lake to a clear path towards compliance.
6. But why views? Views offer a range of beneficial properties that make them flexible and effective for compliance. They are expressive, allowing the representation of multiple policies through tools like projections, filters, UDFs, and joins. They are portable, thanks to SQL’s nature, allowing execution across various engines with minimal adjustments. Views are also modular, serving as drop-in replacements for underlying data; by simply substituting table names with view names while preserving schemas, the same code can operate with additional logic encapsulated within the view. Moreover, views are agile, enabling the deployment of new views or reversion to previous ones with minimal impact. This agility allows for quick bug fixes or policy updates.
7. At the expressivity level, let us demonstrate some key ways in which views can be used to apply policy-specific transformations. For instance, views enable column-level filtering—by excluding certain columns from a table, we can tailor the data presented to the consumer.
8. Views can also be used for column-level masking, where instead of removing a column entirely, we mask or redact the data within it
9. Furthermore, views can implement row-level filters to exclude unqualified rows from results
10. or even perform cell-level masking, where specific data points are obscured based on the individual user’s consent and the data domain.
11. These diverse masking capabilities are not limited to single-view applications; a single table can support multiple views, each representing a different method of data masking and applicable in distinct contexts. We will explore more of this versatility throughout the presentation. 
12. Views are typically stored as metadata in metadata stores like the Hive metastore, and more recently, Iceberg has begun supporting views in its metadata structure. Engines operating on these metadata stores can execute views within their environments, which underscores the importance of portability. At LinkedIn, we leverage tools like Coral for dialect translation, allowing us to define policy views in one SQL dialect and have them executed across any engine, in any other dialect.
13. Let's bring everything together. Imagine we have a data lake containing numerous tables, all stored in a metadata store.
14. The next step involves the policy engine generating a set of views corresponding to those tables. For instance, for Table 1 (T1), we might create three distinct views. Each view represents a separate transformation tailored to a specific use case, hence the view names are labeled as T1UC1 for Use Case 1, and so forth
15. Consequently, each view embodies its unique logic and transformation, operating within a designated context.
16. Now, let’s consider how we can roll out views effectively, given their potent capabilities for ensuring privacy and data protection. One approach is to forcibly change user or application logic by replacing every table name with a new view name in every script. However, this brute force method is not the best approach.
17. It requires a large-scale migration, which can be expensive and slow, and it risks exposing context-specific information in the view names. For example, a table with a business-critical meaning might end up with multiple suffixes, diluting its core significance. If we manually migrate to views, then when a new version of a view is needed, or a new regulation or policy is introduced, we might find the approach unmanageable and not user-friendly.
18. Can we do better? Yes, we can. Here’s where the architecture we refer to as ViewShift comes in. In this system, the user script remains intact, still referring to the table, but during execution, the tables are automatically replaced by the compliance view. For example, let's look at this depiction where the Spark engine attempts to resolve an identifier for a table, such as T1, which originates from the user script. This engine interacts with a metadata store and a catalog connection layer for this purpose. The catalog implementation can then transparently return the corresponding privacy view or an obfuscated view related to the table. Even though the script is written as shown on the left-hand side of the slide, the actual execution simulates as if the user had originally scripted using views.
19. What's particularly advantageous about this approach is its flexibility across different programming languages and platforms. It can be implemented in SQL, Scala, on Trino, or Spark, because the architecture is adaptable to each engine using the same underlying principles.
20. To summarize the overarching benefits of the ViewShift rollout technique—and I will delve into more detailed architecture in the upcoming slides— it's transparent. Users do not need to modify their code to adopt new view names; they continue to use familiar table identifiers. It also supports upcoming policies because new views can be introduced and enforced seamlessly, facilitating easy management of versions, whether for upgrades or rollbacks.
21. Before we explore the ViewShift architecture, let's examine the conventional query engine architecture. Typically, this includes a query engine layer with an internal connector layer, often referred to as the tables and views plugin. Its primary role is to resolve identifiers of tables and views into corresponding objects. When the engine parses and analyzes a query, it submits the identifier to this plugin, which returns the appropriate object for further processing. Normally, a table identifier prompts the return of a table object, and a view identifier prompts a view object.
22. Before the introduction of ViewShift, query engines were typically configured with a Tables and Views Plugin—a fundamental component that interprets table and view identifiers and corresponds them to actual table and view objects within the database. This foundational architecture, which can be likened to a plugin within a larger system, sets the stage for the capabilities of ViewShift. The diagram illustrates a straightforward but critical relationship: when a query is executed, the engine uses this plugin to resolve the names of tables and views to their respective objects, forming the basis for query execution.
23. However, with ViewShift, we've adjusted how table identifiers are resolved. We introduce an additional plugin within the tables and views plugin, tasked with mapping table identifiers to their corresponding view identifiers based on the applicable policy and context. This is especially crucial when a single table identifier may correspond to multiple views, and the appropriate view needs to be selected based on the current context. The context map, part of the View Plugin API, facilitates this by ensuring that alongside the table identifier, a specific view identifier is returned, thus substituting a view object in place of a table object.
24. Compared to the conventional implementation where a table returns a table object and a view returns a view object, our new architecture embeds a transformative plugin that allows table requests to return view objects, thereby seamlessly integrating privacy by default through ViewShift.
25. Now, I will hand over the presentation to Khai, who will discuss an end-to-end use case that leverages ViewShift for a recent compliance initiative at LinkedIn. Over to you, Khai.