Migrating legacy ERP data into Hadoop

Editor's Notes

1. Hello, and welcome to “Migrating Legacy ERP Data into Hadoop”. This is a technical presentation for Dataworks 2018 in San Jose, CA.
2. Today’s illuminating speakers are Jordan Martz, Director of Technology Solutions for Attunity. And, David Freriks, Technology Evangelist, at Qlik.
3. ERP data can be hard to interact with at the database level and translating logic from your business from tables can be even harder. Legacy ERP architecture constructs offer a system that is a great for ERP, but a real challenge to get business insights from. As technology partners, Qlik, Attunity and Hortonworks offer a technology solution that helps you get large-scale ERP data to a platform where you can perform analytics. This joint solution makes ERP data available for business users who want to extract value from it. This session explains and demonstrates how bottlenecks within the ERP infrastructure are removed when business use Attunity Replicate to ingest onto the Hortonworks Data Platform where Qlik software is used to begin building applications for analytics.
4. Today, we’re going to discuss a few use cases and product scenarios that make up a solution that we call “SMASH”. To keep this from turning into alphabet soup, let’s explain a few things. Qlik Sense: a highly flexible and scalable analytics platform for BI Microsoft Azure: set of cloud services to help organizations meet their business challenges. Attunity Replicate: software that accelerates data replication, ingest and streaming across a wide range of heterogeneous databases, data warehouses and data platforms SAP: data management platforms to handle both transactions and analytics in memory on a single data copy Hortonworks Data Flow (HDF): end-to-end platform that collects, curates, analyzes and acts on data in real-time with a drag-and-drop visual interface
5. To set the stage, and because we’re presenting at the Hortonworks Dataworks conference, let’s start by talking about the Hortonworks Data Platform (HDP) and Hortonworks Data Flow (HDF). Together, they form the Connected Data Platform that works with Data in Motion (connected, real-time, tracked) and Data at Rest (massive scale analysis, retention, security). Modern Data Applications are built on the Connected Data Platform.
6. SAP is a legacy ERP application and it’s the one that we’ll focus on for today’s presentation.
7. The Raw content is the direct result of the Replicate tasks. Attunity delivers a semantic layer of source objects from the SAP application so you can choose individual transactions for replication. It is not necessary to know or understand the underlying SAP data model.
8. Compressed Data Models are a flattening of the SAP data model. In the SAP application, data is spread out over dozens or hundreds of tables, views, aggregate or indexed tables. Attunity can compress those structures into a handful of objects/tables where all of the source data is available, but reduced to fewer objects. These compressed objects are still organized by individual SAP object or document (examples below): Sales Documents Delivery Documents Billing Documents Finance Documents Customer Masters Material Masters
9. SAP Object Logical Views Logical Views are views created to define and display a particular function of the business. For example, the Sales Order (Order to Cash) process is made up of various SAP documents (customer & material master data, order, deliveries, and billing documents). The logical view takes to the most relevant aspects of those processes and provides a Logical View that can be used to query and report on. One of the unique aspects of the Logical View is that check table relationships are included so metadata can be linked to the test (i.e. Company Code 1000 can be reports as it’s text value “North America – US”). Exmaples of preconfigured Logical View are below. This list can be easily extend and developed through a Professional Services deliverable: Controlling Project Systems General Ledger Accounts Payable Accounts Receivable HR Purchasing (Procurement) Production Planning Sales/Logistics
10. SAP Object Logical Views Logical Views are views created to define and display a particular function of the business. For example, the Sales Order (Order to Cash) process is made up of various SAP documents (customer & material master data, order, deliveries, and billing documents). The logical view takes to the most relevant aspects of those processes and provides a Logical View that can be used to query and report on. One of the unique aspects of the Logical View is that check table relationships are included so metadata can be linked to the test (i.e. Company Code 1000 can be reports as it’s text value “North America – US”). Exmaples of preconfigured Logical View are below. This list can be easily extend and developed through a Professional Services deliverable: Controlling Project Systems General Ledger Accounts Payable Accounts Receivable HR Purchasing (Procurement) Production Planning Sales/Logistics
11. The Landscape There are three very important BI constituencies in the Big Data space. Data Scientists are one group. While very few exist in most organizations, they are very specialized in their data mining and advanced analytical skills with data. Data Knowledge Workers, like Actuaries, Financial Planners and Statisticians are a larger group, but still number in the dozens in a large organization. By far, the largest group are the Business Analysts, which can number in the hundreds, or even thousands in large organizations. These users are the daily BI users who analyze departmental and corporate data to understand and act upon metrics and discoveries that impact their business. They can be managers, analysts, specialists, executives or SMEs for a departmental group. But they all know their business area well and consume a lot of BI already. The Problem Most BI tools claiming to meet the needs of Big Data will only concentrate on the Data Scientists’ needs. While these are important needs, they typically serve very few people in an organization, and leave most of the people with Big data needs out in the cold. How is Qlik Different? Qlik’s platform is an in-memory-first model, with the ability to reach out to databases directly for deep drilling. What this means is that the bulk of all Big Data analytics (wide, exploratory BI) will remain in-memory at sub-second response times, and only when deep drilling is needed will Qlik reach out to databases for the more expensive and process-intensive queries for deeper drilling needs. This mix matches the usage patterns that organizations need. Will you opt for a BI tool that specializes in the needs of the few? Or opt for a platform that satisfies all needs, with a comprehensive Big Data approach?